Reviews in this chain:
└#378 feat(raycicmd): add matrix expansion types and selector resolution
　└#379 feat(raycicmd): integrate matrix expansion into pipeline conversion

Summary of Changes

Hello @andrew-anyscale, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the CI/CD pipeline's flexibility and efficiency by implementing dynamic matrix expansion and selective dependency management. It allows for the generation of specific pipeline steps based on matrix configurations and enables subsequent steps to depend only on relevant matrix variants. This improvement streamlines the build process by preventing the execution of redundant steps and provides finer control over pipeline dependencies.

Highlights

• Matrix Expansion: Introduced the capability to expand matrix-defined steps within the pipeline, generating individual steps for each matrix variant.
• Selective Dependency: Enabled pipeline steps to selectively depend on a subset of matrix-defined steps, allowing for more granular control and efficient execution.
• Tag Filtering Integration: Integrated tag filtering with matrix expansion, allowing unnecessary matrix variants to be filtered out, reducing overall build time.
• New Configuration and Tests: Added a new YAML configuration for matrix selection tests and comprehensive unit tests to validate the matrix expansion and filtering logic.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

expandedStepRaw := inst.substituteValues(step)
expandedStep, ok := expandedStepRaw.(map[string]any)
if !ok {
    return nil, nil, fmt.Errorf("matrix step %q: substitution returned unexpected type", baseKey)
}

matches, err := expandMatrixSelector(sel, ctx.registry, ctx.expandedKeys)
if err != nil {
    return nil, err
}
if len(matches) == 0 {
    return nil, fmt.Errorf("matrix selector for key %q matched no instances", sel.Key)
}
result = append(result, matches...)

depends_on:
  - key: build-step
    matrix:
      python: "3.99"  # non-existent variant

gitar auto-apply:on         

gitar display:verbose         

_{Was this helpful? React with 👍 / 👎 | Gitar}

Example:

group: matrix selection test
tags:
  - matrix-test
steps:
  # Matrix step with 5 values - only "selected-1" and "selected-2" should be run
  # when filtered (others are skipped)
  - label: "Matrix {{matrix.variant}}"
    key: matrix-step
    command: |
      echo "Running variant: {{matrix.variant}}"
      if [ "{{matrix.variant}}" != "selected-1" ] && [ "{{matrix.variant}}" != "selected-2" ]; then
        echo "ERROR: This variant should not have been spawned!"
        echo "Only 'selected-1' and 'selected-2' variants should run when matrix-test tag is used."
        exit 1
      fi
      echo "Correct variant spawned."
    depends_on: forge
    matrix:
      setup:
        variant:
          - "selected-1"
          - "selected-2"
          - "skipped-1"
          - "skipped-2"
          - "skipped-3"

  # This step depends on only the "selected" variants
  - label: "Depends on selected only"
    key: depends-on-selected
    command: echo "This step depends on only the selected variants"
    depends_on:
      - key: matrix-step
        matrix:
          variant: ["selected-1", "selected-2"]
      - forge

  - label: "Final validation"
    tags: always
    key: final-validation
    command: echo "Final validation"
    depends_on: depends-on-selected